Using Reinforcement Learning to Attenuate for Stochasticity in Robot Navigation Controllers

International audienceBraitenberg vehicles are bio-inspired controllers for sensor-based local navigation of wheeled robots that have been used in multiple real world robotic implementations. The common approach to implement such non-linear control mechanisms is through neural networks connecting sensing to motor action, yet tuning the weights to obtain appropriate closed-loop navigation behaviours can be very challenging. Standard approaches used hand tuned spiking or recurrent neural networks, or learnt the weights of feedforward networks using evolutionary approaches. Recently, Reinforcement Learning has been used to learn neural controllers for simulated Braitenberg vehicle 3a-a bio-inspired model of target seeking for wheeled robots-under the assumption of noiseless sensors. Real sensors, however, are subject to different levels of noise, and multiple works have shown that Braitenberg vehicles work even on outdoor robots, demonstrating that these control mechanisms work in harsh and dynamic environments. This paper shows that a robust neural controller for Braitenberg vehicle 3a can be learnt using policy gradient reinforcement learning in scenarios where sensor noise plays a non negligible role. The learnt controller is robust and tries to attenuate the effects of noise in the closed-loop navigation behaviour of the simulated stochastic vehicle. We compare the neural controller learnt using Reinforcement Learning with a simple hand tuned controller and show how the neural control mechanism outperforms a naïve controller. Results are illustrated through computer simulations of the closed-loop stochastic system

Adaptive reinforcement learning with active state-specific exploration for engagement maximization during simulated child-robot interaction

International audienceUsing assistive robots for educational applications requires robots to be able to adapt their behavior specifically for each child with whom they interact. Among relevant signals, non-verbal cues such as the child's gaze can provide the robot with important information about the child's current engagement in the task, and whether the robot should continue its current behavior or not. Here we propose a reinforcement learning algorithm extended with active state-specific exploration and show its applicability to child engagement maximization as well as more classical tasks such as maze navigation. We first demonstrate its adaptive nature on a continuous maze problem as an enhancement of the classic grid world. There, parame-terized actions enable the agent to learn single moves until the end of a corridor, similarly to "options" but without explicit hierarchical representations. We then apply the algorithm to a series of simulated scenarios, such as an extended Tower of Hanoi where the robot should find the appropriate speed of movement for the interacting child, and to a pointing task where the robot should find the child-specific appropriate level of expressivity of action. We show that the algorithm enables to cope with both global and local non-stationarities in the state space while preserving a stable behavior in other stationary portions of the state space. Altogether, these results suggest a promising way to enable robot learning based on non-verbal cues and the high degree of non-stationarities that can occur during interaction with children

On the Convergence of Braitenberg Vehicle 3a immersed in Parabolic Stimuli

Abstract-Braitenberg vehicles are well known models of animal behaviour prone to be used as a mobile robot controller. Due to the qualitative simplicity of their behaviour they are used for teaching robotics, whilst the lack of a quantitative theory makes its use for research purposes troublesome. This paper contributes to our formal understanding of Braitenberg vehicle 3a by presenting some convergence properties of its trajectories under parabolic shaped stimuli or potential functions. We show new features of the vehicle motion unreported in robotics like, their conditional stability, their oscillatory behaviour and the existence of a preferred convergence direction. Quantitatively identifying the behaviour of Braitenberg vehicles allows to use them in robotics on a sound basis, not just relying on incomplete qualitative understanding as done by earlier works

On The Application Of Colour Histograms For Mobile Robot Localisation

In this paper we present an appearance-based method to be used for topologically localising the robot. The information extracted from the image sequences is just an approximation of the colour probability density function estimated using a non-parametric clustering paradigm, the Self-organising Map neural network, together with the information obtained from the segmentation of a single image, i.e the pixel ratio in each cluster. The compactness of the stored information for each view allows to store multiple references and to compute the localisation in real time

Active Landmark Perception

Vision based landmark identification can be greatly improved by means of dynamically adjusting camera's parameters to select scene regions that could contain favorable data. This paper present an approach for sonar-based dynamic selection of tilt and zoom values. Data take

Adaptive reinforcement learning with active state-specific exploration for engagement maximization during simulated child-robot interaction

Using assistive robots for educational applications requires robots to be able to adapt their behavior specifically for each child with whom they interact.Among relevant signals, non-verbal cues such as the child’s gaze can provide the robot with important information about the child’s current engagement in the task, and whether the robot should continue its current behavior or not. Here we propose a reinforcement learning algorithm extended with active state-specific exploration and show its applicability to child engagement maximization as well as more classical tasks such as maze navigation. We first demonstrate its adaptive nature on a continuous maze problem as an enhancement of the classic grid world. There, parameterized actions enable the agent to learn single moves until the end of a corridor, similarly to “options” but without explicit hierarchical representations.We then apply the algorithm to a series of simulated scenarios, such as an extended Tower of Hanoi where the robot should find the appropriate speed of movement for the interacting child, and to a pointing task where the robot should find the child-specific appropriate level of expressivity of action. We show that the algorithm enables to cope with both global and local non-stationarities in the state space while preserving a stable behavior in other stationary portions of the state space. Altogether, these results suggest a promising way to enable robot learning based on non-verbal cues and the high degree of non-stationarities that can occur during interaction with children